Structural Regular Expressions

The current UNIX® text processing tools are weakened by the built-in concept of a line. There is a simple notation that can describe the ‘shape’ of files when the typical array-of-lines picture is inadequate. That notation is regular expressions. Using regular expressions to describe the structure in addition to the contents of files has interesting applications, and yields elegant methods for dealing with some problems the current tools handle clumsily. When operations using these expressions are composed, the result is reminiscent of shell pipelines. The Peter-On-Silicon Problem In the traditional model, UNIX text files are arrays of lines, and all the familiar tools — grep, sort, awk, etc. — expect arrays of lines as input. The output of ls (regardless of options) is a list of files, one per line, that may be selected by tools such as grep: ls -l /usr/ken/bin | grep ’rws.*root’ (I assume that the reader is familiar with the UNIX tools.) The model is powerful, but it is also pervasive, sometimes overly so. Many UNIX programs would be more general, and more useful, if they could be applied to arbitrarily structured input. For example, diff could in principle report differences at the C function level instead of the line level. But if the interesting quantum of information isn’t a line, most of the tools (including diff) don’t help, or at best do poorly. Worse, perverting the solution so the lineoriented tools can implement it often obscures the original problem. To see how a line oriented view of text can introduce complication, consider the problem of turning Peter into silicon. The input is an array of blank and non-blank characters, like this: ####### ######### #### ##### #### #### # #### ##### #### ### ######## ##### #### ######### #### # # #### ## # ### ## ### # ### ### ## ## # # #### # # ## # ## The output is to be statements in a language for laying out integrated circuits: rect minx miny maxx maxy The statements encode where the non-blank characters are in the input. To simplify the problem slightly, the coordinate system has x positive to the right and y positive down. The output need not be efficient in its use of rectangles. Awk is the obvious language for the task, which is a mixture of text processing and geometry, hence arithmetic. Since the input is an array of lines, as awk expects, the job should be fairly